feat(review): backtest confidence-threshold changes against real signal history in ORB's live gate#8142
Conversation
…al history in ORB's live gate When a PR edits a known AI-judgment confidence threshold, ORB now detects the change, replays it against recorded rule-fired/human-override history via the calibration engine, and folds the precision/recall comparison into the existing unified PR comment. Runs inline in the review pipeline using ORB's own diff and D1 access — no separate CI job or bot comment.
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | 2ba9f33 | Commit Preview URL Branch Preview URL |
Jul 23 2026, 02:12 AM |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #8142 +/- ##
==========================================
- Coverage 92.03% 92.02% -0.01%
==========================================
Files 760 763 +3
Lines 77424 77496 +72
Branches 23399 23420 +21
==========================================
+ Hits 71255 71318 +63
- Misses 5061 5062 +1
- Partials 1108 1116 +8
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
Caution 🛑 LoopOver review result - fixes requiredReview updated: 2026-07-23 02:25:23 UTC
Review summary Nits — 6 non-blocking
CI checks failing
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionPartially addressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
…tory in a dedicated CI job (#8139) A PR that rewrites detection logic (not just a threshold, #8138/#8142) now gets the same honest before/after backtest: a new path-filtered workflow checks out the PR's head AND base, replays linked_issue_scope_mismatch's captured raw context (#8129/#8130 + the model-response capture) through both versions of buildLinkedIssueSatisfactionResult, scores them with @loopover/engine's scoreBacktest/compareBacktestScores, posts its own advisory PR comment, and persists a calibration.logic_backtest_run audit event for #8140's track record. backtest-track-record.ts now aggregates both sibling event types, so #8105's Phase-2 decision reads threshold AND logic runs, not a partial record. Runs in CI, not ORB's Worker, because verifying a logic rewrite requires executing the PR's own code -- the same trust boundary validate-code already uses, and one the credential-holding Worker must never cross. Pure core (registry scoped to linked_issue_scope_mismatch, secret_leak permanently excluded per #8130, classify construction, comment/SQL rendering) is fully unit-tested; the CLI is thin IO glue per the export-d1-data.ts precedent. Advisory only -- never a required check, never blocks merge (#8105).
…tory in a dedicated CI job (#8147) * feat(review): capture the model's raw response in linked_issue_scope_mismatch's fired-event metadata (#8139) The existing #8129 raw-context capture (issueText/prTitle/prBody/diff) rebuilds the prompt but never stored what the model actually returned, so a future logic backtest couldn't replay parseLinkedIssueSatisfactionOpinion/ buildLinkedIssueSatisfactionResult against real history -- only the prompt inputs, not the output they need to re-parse. Threads the raw text through runWorkersSatisfactionOpinion and the BYOK path, bounded at the capture site in processors.ts the same way the other raw-context fields are. Part of #8139 (logic/regex backtest CI check); the CI job itself is not yet implemented. * feat(ci): backtest logic/regex detection changes against recorded history in a dedicated CI job (#8139) A PR that rewrites detection logic (not just a threshold, #8138/#8142) now gets the same honest before/after backtest: a new path-filtered workflow checks out the PR's head AND base, replays linked_issue_scope_mismatch's captured raw context (#8129/#8130 + the model-response capture) through both versions of buildLinkedIssueSatisfactionResult, scores them with @loopover/engine's scoreBacktest/compareBacktestScores, posts its own advisory PR comment, and persists a calibration.logic_backtest_run audit event for #8140's track record. backtest-track-record.ts now aggregates both sibling event types, so #8105's Phase-2 decision reads threshold AND logic runs, not a partial record. Runs in CI, not ORB's Worker, because verifying a logic rewrite requires executing the PR's own code -- the same trust boundary validate-code already uses, and one the credential-holding Worker must never cross. Pure core (registry scoped to linked_issue_scope_mismatch, secret_leak permanently excluded per #8130, classify construction, comment/SQL rendering) is fully unit-tested; the CLI is thin IO glue per the export-d1-data.ts precedent. Advisory only -- never a required check, never blocks merge (#8105). * fix(ci): fail the logic backtest open — an advisory job must never red-X a PR on infra errors (#8139) First live run surfaced two problems at once: the repo's CLOUDFLARE_API_TOKEN is scoped Workers Scripts:Edit only (no D1 access), so the corpus export died with Cloudflare 7403 -- and that red check is not harmless, because the review engine auto-closes contributor PRs on ANY failed check, required or not. An advisory job that can go red on an under-scoped token, a D1 outage, or a comment-post hiccup would let our own plumbing close innocent contributor PRs touching the watched paths. Every D1/comment step now catches its own failure, emits a ::notice explaining exactly what was skipped and why, and keeps the job green -- 'never blocks merge' (#8105) now holds against the job's own failures, not just its verdicts. Enabling the backtest for real needs a D1-scoped token in the CLOUDFLARE_API_TOKEN secret; until then the job is a visible no-op. * feat(ci): persist the corpus checksum with each logic-backtest run as its reproducibility freeze point (#8139) The persisted run recorded the comparison and both shas but not WHICH corpus snapshot it scored -- without that, a skeptical re-run can't prove it replayed the same history. The manifest's own checksum (#8084) is already in hand at run time; thread it into the persisted metadata and the PR comment, so checksum + head/base shas + the public scoring code make every run independently re-runnable end to end (the reproducible-backtest posture #8136 is evaluating).
…#8114) (#8153) #8138 (threshold backtest, shipped ORB-native in #8142) and #8139 (logic backtest CI check, shipped in #8147) are exactly the #3047 failure shape -- new CLI flags, a new CI check with its own trigger/comment behavior, a gate-adjacent pending policy decision (#8105) -- with no docs. Adds content/docs/backtest-calibration.mdx (corpus model, the two backtest mechanisms and why they run where they run, how to read the Pareto-floor comparison, secret_leak's permanent exclusion, the track-record/corpus-export CLIs, and the advisory-only guarantee with #8105 noted as pending) plus the thin route, nav/command-palette entries, and the regenerated route tree. The two pinned docs-route-count tests move 48 -> 49.
Summary
LINKED_ISSUE_SATISFACTION_CONFIDENCE_FLOOR,DEFAULT_AI_REVIEW_CLOSE_CONFIDENCE), replays the change against real recorded rule-fired/human-override history, and folds a precision/recall comparison into the existing unified PR comment.Architecture note (differs from #8138's literal text)
#8138 as originally scoped described a standalone CI workflow (
.github/workflows/*.yml) shelling out towrangler d1 execute. During implementation that approach was replaced with an ORB-native one: the same detection/backtest/render logic now lives insrc/services/threshold-backtest.ts(pure) andsrc/services/threshold-backtest-run.ts(D1 orchestration via the existingSignalStore), called directly fromsrc/queue/processors.tsand threaded into the unified comment viaunified-comment-bridge.ts→unified-comment.ts. This avoids a second CI job and a second bot comment on an already-slow CI pipeline, and keeps threshold backtesting behind the same live gate infra everything else uses. All functional requirements from the issue (detect the two named constants, backtest against real history, render a comparison, persist the structured result, never block merge) are preserved.Test plan
npm run typecheck— cleanthreshold-backtest.test.ts,threshold-backtest-run.test.ts,threshold-backtest-advisory-wiring.test.ts,backtest-threshold-engine.test.ts— 100% coverage (statements/branches/functions/lines) on both newsrc/services/filespackages/loopover-engineinternal suite — 655/655 passing (barrel-export additions don't regress anything)npm run test:ciCloses #8138